Method and apparatus for preventing overloads of power distribution networks

ABSTRACT

Systems and methods for monitoring power in power distribution systems are provided. In one aspect, a system for monitoring power includes a power monitoring device that measures a value of at least one characteristic of power provided to a branch of a power distribution system. The power monitoring device includes an output that provides the value measured. The system further includes a controller having an input to receive the value measured and an output that couples to a first device powered by the branch to send a maximum power signal to the first device to command the first device to operate at a percentage of maximum power.

RELATED APPLICATIONS

[0001] This application is a continuation of U.S. patent applicationSer. No. 10/038,701, filed on Jan. 2, 2002, which is incorporated hereinby reference.

FIELD OF THE INVENTION

[0002] The present invention relates generally to methods and apparatusfor measuring current or power delivered to loads in power distributionnetworks, and more specifically, to methods and apparatus for preventingoverload conditions in power distribution equipment used to powerequipment having variable input power requirements.

BACKGROUND OF THE INVENTION

[0003] The proliferation of the Internet has created a need for largescale data centers that contain tens, if not hundreds, of racks ofcomputing equipment, such as servers and routers. One of the majorproblems confronted by designers of these data centers is therequirement to route facility power to each of these racks of equipment.Typically, branch circuits from a primary or a secondary distributionpanel are routed to groups of racks to provide power to the equipment inthe rack. Each of the branch circuits is designed to provide apredetermined maximum power level or current, and the size of cablingused to route the power for a branch, and the size of circuit breakersused for the branch are selected based on this predetermined maximumpower level or, more typically, predetermined maximum current.

[0004] Typically, it is desirable to design each of the branch circuitssuch that the total current drawn by the equipment coupled to any givenone of the branch circuits is at some predetermined percentage (forexample 50%) of the maximum current level for that branch circuit. Thisallows some flexibility to add additional equipment to racks andprovides a safety margin below the maximum current level.

[0005] To properly design the routing of the branch circuits, it isdesirable to know, with some accuracy, the current that is drawn by theequipment in the racks. Traditionally, the power or current drawn bycomputer equipment could be determined based on manufacturers'specifications and/or by making actual measurements of the current beingdrawn by the equipment.

[0006] These measurements and specifications are only useful forequipment for which the current draw is substantially static, which inthe past was true for typical computing equipment. However, for newercomputing equipment, the current draw is typically not static due to anumber of factors including: 1) many computers utilize some form ofpower management strategy which minimizes the power (and current)consumption of the computer by turning off or slowing down subsystemswithin the computer when they are not in use; 2) cooling systems (i.e.,fans) are often speed controlled based on air and component temperaturesto reduce power consumption and acoustic noise generation; and 3) theamount of power drawn by the processors and memory systems in computershas increased steadily with the increase of speed of the processors, sothat the power consumed by the processors and memory subsystems mayexceed 50% of the total power draw of a computer. The power drawn byprocessors and memory systems is variable depending on the processingload, and since the total power of these systems may be a significantportion of the total power, the total power draw of a computer can varysignificantly depending on the processing load on the computer.

[0007] The operating systems of most computers are capable ofsimultaneously performing multiple tasks by assigning segments of theCPU processing time to each of the tasks on a priority basis. Anyremaining segments of the CPU processing time are occupied by an idletask in which the CPU can be halted and all associated clocks can bestopped to reduce the power draw of the computer. Further, somecomputers, for example, those that utilize the Windows® 98 or Windows®2000 operating system, have an Advanced Control and Power Interface(ACPI) feature that allows the operating system to control power to fansand other devices in the computer to further reduce the power drawn bythe computer. Because of the factors described above, it is not unusualfor a more modern system to consume twice as much power when theprocessors are fully computationally loaded and operating in a warmenvironment, then when computationally idle and operating in a coolenvironment.

[0008] The variability of the power draw of computers complicates theelectrical design of data centers. Computer manufacturers typicallyprovide power ratings on nameplates. These nameplate values aretypically maximum values that are determined based on the maximum powerthat a computer may draw when fully loaded with all options and with allsubsystems operating at full load. Because of conservative approachestaken in determining nameplate values, they are often greater than evenworst case values for a given computer, and accordingly are of littleuse to an electrical facility designer. While a designer may measure thecurrent drawn by a computer or a set of computers to determine the powerdraw, it is typically not known at the measurement time, whether thecomputer is at full load or at what percentage of full load the computeris operating.

[0009] Several problems may occur when circuit branches are designedbased on measured power draw values of computers. First, the wiring usedin power routing circuits may be undersized for full load conditions,and second, when one or more of the computers powered from a branch areoperated at full load, the current drawn may exceed the circuit breakervalue for the branch, causing the circuit breaker to trip and disconnectpower to the computers. For critical applications of computers, any suchpower interruption is often unacceptable. Further, to prevent powerinterruptions to critical computers, it is common to use uninterruptiblepower supplies (UPSs) for these computers. Often, one UPS is used topower multiple computers or racks of computers. To properly size theUPS, it is necessary to know the power draw of each of the computers andother equipment powered by the UPS. The variability of the power draw innewer computers makes it difficult to properly size a UPS for theseapplications.

SUMMARY OF THE INVENTION

[0010] Embodiments of the present invention provide improved systems andmethods for measuring the current or power draw of computers and racksof equipment that overcome problems described above.

[0011] A first aspect of the present invention is directed to a systemfor monitoring power in a power distribution system. The system includesa power monitoring device located in the power distribution system tomeasure a value of at least one characteristic of power provided to abranch of the power distribution system, the power monitoring devicehaving an output that provides the value measured, and a controllerhaving an input to receive the value measured and an output that couplesto a first device powered by the branch of the power distribution systemto send a power signal to the first device to command the first deviceto operate at a predetermined percentage of maximum power.

[0012] The system for monitoring power can further include a pluralityof power monitoring devices, each located in the power distributionsystem to measure at least one characteristic of power provided to arespective branch of the power distribution system, and each having anoutput to couple to the controller to provide a value of thecharacteristic measured. Each of the respective branches of the powerdistribution system can provide power to at least one respective device,and the controller can be adapted to send a power signal to eachrespective device to command each device to operate at the predeterminedpercentage of maximum power. The controller can be adapted to send thepower signal to devices powered by one branch at a same time, to causeeach of the devices on the one branch to operate at the predeterminedpercentage of maximum power. The controller can be adapted to total thevalues measured for each of a plurality of branch circuits and comparethe total with a first overload value to detect an overload condition.The controller can be adapted to send an alarm signal to an operatorupon detection of an overload condition. The controller can be adaptedto send a signal to disconnect power to one or more devices upondetection of an overload condition. The at least one characteristic canbe electrical current.

[0013] The controller of the power monitoring system can further includea first network interface to communicate with devices powered by thepower distribution system over a first communications network and asecond network interface to communicate over a second communicationsnetwork. Each of the plurality of power monitoring devices can include anetwork interface to communicate with the controller over the secondcommunications network. The power distribution system can include anuninterruptible power supply, and the controller can be adapted tocommunicate with the uninterruptible power supply to detect that theuninterruptible power supply is operating on battery mode and replacethe first overload value with a second overload value. The controllercan be adapted to send a signal to interrupt power to at least onedevice upon detection that the uninterruptible power supply is operatingon battery mode. The system can further include a plurality oftemperature sensors that monitor temperature at locations within afacility, each of the temperature sensors having an output tocommunicate a temperature value to the controller. The controller can beadapted to compare temperature values received from the temperaturesensors with predetermined values to detect an over temperature errorcondition, and upon detection of an over temperature error conditionsend an alarm signal. The controller can be adapted to send a signal tointerrupt power to at least one device upon detection of an overtemperature error condition. The predetermined percentage of maximumpower can be 100 percent.

[0014] Another aspect of the present invention is directed to a methodfor monitoring and controlling a power distribution system that has aplurality of circuit branches for providing power to a plurality ofdevices. The method includes controlling a first device on a firstcircuit branch to operate at a predetermined percentage of maximumpower, detecting a first value for a characteristic of power provided tothe first circuit branch, controlling a second device on a secondcircuit branch to operate at a predetermined percentage of maximumpower, detecting a second value for a characteristic of power providedto the second circuit branch, adding the first value to the second valueto obtain a total value, comparing the total value to an overload valueto detect an overload condition, and indicating an alarm condition whenthe total value exceeds the overload value.

[0015] The first device can be controlled to operate at less than thepredetermined percentage of maximum power when the second device iscontrolled to operate at the predetermined percentage of maximum power.The method can further include controlling one of the plurality ofdevices to operate in a reduced power mode upon detection of an overloadcondition. The method can further include interrupting power to one ofthe plurality of devices upon detection of an overload condition. Thecharacteristic measured can be electrical current. The method canfurther include communicating with the first device and the seconddevice over a first communications network, and communicating with powerdetection devices over a second communications network. The powerdistribution system can further include an uninterruptible power supply,and the method can further include detecting when the uninterruptiblepower supply is operating in a battery mode, and controlling at leastone of the plurality of devices to operate in a reduced power mode upondetection of the battery mode. The method can further includeinterrupting power to at least one of the plurality of devices upondetection of the battery mode. The power distribution system can be atleast partially contained within a facility, and the method can furtherinclude measuring air temperature at a plurality of locations within thefacility, comparing at least one value of air temperature measured witha predetermined value to detect an over temperature condition, andcontrolling at least one of the plurality of devices to operate in areduced power mode upon detection of the over temperature condition. Thepredetermined percentage of maximum power can be 100 percent.

[0016] Yet another aspect of the present invention is directed to asystem for monitoring and controlling a power distribution system thathas a plurality of circuit branches for providing power to a pluralityof devices. The system includes means for controlling each of theplurality of devices to operate at a predetermined percentage of maximumpower, and means for detecting a value of a characteristic of powerprovided to each of the plurality of circuit branches.

[0017] The system can further include means for comparing a total valueof the characteristic with a predefined value to detect an overloadcondition. The system can further include means for interrupting powerto at least one of the plurality of devices when an overload conditionis detected. The characteristic can be electrical current. The powerdistribution system can include at least one uninterruptible powersupply, and the system can further include means for detecting that theuninterruptible power supply is in a battery mode of operation, andmeans for adjusting the predefined value when the uninterruptible powersupply is in the battery mode of operation. The system can furtherinclude means for detecting air temperature values in a facilitycontaining the power distribution system. The system can further includemeans for comparing the detected air temperature values withpredetermined temperature values, and means for interrupting power to atleast one of the plurality of devices when the detected air temperaturevalues exceed the predetermined temperature values. The predeterminedpercentage of power can be 100 percent.

BRIEF DESCRIPTION OF THE DRAWINGS

[0018] For a better understanding of the present invention, reference ismade to the drawings which are incorporated herein by reference and inwhich:

[0019]FIG. 1 shows a typical layout of the power distribution system ina data center;

[0020]FIG. 2 shows a power distribution control system in accordancewith a first embodiment of the present invention;

[0021]FIG. 3 shows the power distribution control system of FIG. 2operatively coupled to the power distribution system of FIG. 1.

[0022]FIG. 4 shows a flow chart of a method of controlling power flow ina power distribution system in accordance with one embodiment of thepresent invention;

[0023]FIG. 5 shows a power distribution control system of a secondembodiment of the present invention; and

[0024]FIG. 6 shows a power distribution control system of a thirdembodiment of the present invention.

DETAILED DESCRIPTION

[0025] Embodiments of the present invention that provide methods andsystems for monitoring and controlling power distribution in datacenters will now be described. As understood by those skilled in theart, embodiments of the present invention are not limited for use indata centers, but may also be used in other facilities in which it isdesired to monitor and control power distribution. Further, embodimentsof the present invention may also be used aboard ships, airplanes orother mobile platforms where it is desired to monitor and control powerdistribution.

[0026]FIG. 1 provides a diagram of a typical power distribution system100 for a data center. The power distribution system 100 includes afacility input power port 102, an uninterruptible power supply (UPS)104, a power distribution unit (PDU) 106, a 108, and three powerdistribution branches 110, 112 and 114. In FIG. 1, the powerdistribution system is used to power nine loads 116 a-116 i. As is knownto those skilled in the art, some of the components shown for the powerdistribution system are optional. For example, the UPS is an optionalcomponent that is used to provide power to the loads in the event ofdisruption of the facility power.

[0027] The PDU 106 may include a number of power devices such asswitches, a transformer and may include circuit breakers in addition toor in place of circuit breakers contained in the circuit breaker panel108. The circuit breaker panel 108 distributes power from the PDU toeach of the three power distribution branches 110, 112 and 114, andprovides circuit breaker protection for each of the power distributionbranches. The power loads 116 a-116 i may be equipment racks containingmultiple computers, standalone computers, standalone mass storagedevices, or any other equipment that is typically found in a datacenter.

[0028] Embodiments of the present invention provide power monitoring andcontrol for power distribution systems such as power distribution system100. FIG. 2 provides a block diagram of a power monitoring and controlsystem 200 of the present invention. The system includes a centralcontroller 201, a first set of power monitoring devices 202A, 202B, and202C, a second set of power monitoring devices 204A, 204B and 204C, anintelligent power strip 205, a consolidator 206, a first computer 208, asecond computer 210, and a network 211 operatively coupling thecomponents of the system 200.

[0029] With the exception of containing a power control module 212, thefirst computer and the second computer are standard computers that aretypically found in a data center and may be functioning as servers,routers or in some other capacity, and may be mounted in racks of thedata center. In different embodiments of the present invention, thepower control module 212 is implemented using software, hardware, or acombination of software and hardware. In one embodiment the powercontrol module is configured to respond to a signal received over thenetwork 211 and control the computer in which it resides to consumemaximum power by ensuring that all of the subsystems within the computerare operating at substantially 100%, all cooling systems are operatingat 100%, and the computational load on all processors is at 100%. Inother embodiments, the power can be controlled at a predeterminedpercentage of maximum power.

[0030] In some embodiments of the present invention, the power controlmodule is designed to increase the power draw of a computer to maximum,without substantially interfering with tasks being performed by thecomputer. In particular, in one embodiment of the present invention,that will now be described, the power control module is configured tooperate with computers that conserve power by utilizing an idle task(described in the background section above). For this embodiment, thepower control module creates an additional task that is assigned apriority level lower than all other tasks being processed by the CPU,but higher than the idle task. The additional task is designed to fullyutilize the CPU and all disk drives, memories and other devices withinthe computer for the entire idle time of the CPU to maximize powerconsumption by the computer. To ensure that the additional task has theappropriate priority, the control module may reconfigure the prioritiesof each of the other tasks. For computers that have multiple processingunits, each of the processing units is configured to operate at maximumcapacity. By operating as described above, the control module is able tocause a computer to operate at full, or near full power, withoutaffecting the operation of tasks being performed by the computer.Further, in one particular embodiment, for use with computers thatutilize ACPI or other similar means to control power draw of thecomputers, the power control module may be configured to control thespeed of fans and the operation of other devices to operate at maximumpower when desired.

[0031] Each of the power monitoring devices of the first set 202A, 202Band 202C and each of the power monitoring devices of the second set204A, 204B, 204C are inline power monitoring devices that, in oneembodiment, measure the current of a particular power feed. In otherembodiments, the power monitoring devices may directly measure power.The power monitoring devices of the first set are intelligent devicesthat have internal network interface circuitry to enable the devices tocommunicate with other devices, such as the controller 201, over thenetwork 211. The power monitoring devices of the second set have lessintelligence than the devices of the first set and do not have theability to communicate directly with the network 21 1. In oneembodiment, the monitoring devices of the second set utilize point topoint signaling such as RS-232 to communicate power levels to theconsolidator 206. In other embodiments, the monitoring devices of thesecond set may utilize a network scheme or bus scheme like an RS-485multi-drop bus, a power line carrier network, a Controller Area Network(CAN) bus, or a LONWORKS® twisted pair network, to communicate with theconsolidator.

[0032] The consolidator 206 has multiple logical inputs to receive thesignals from each of the power monitoring devices of the second set. Inaddition, the consolidator has a network interface to allow it tocommunicate with the network 211. In embodiments of the presentinvention, the consolidator receives data indicative of power levelsmeasured by each of the power monitoring devices coupled to it andforwards these levels to the controller 201 over the network, along withidentifying information for each of the devices. In one embodiment, theconsolidator is a rack mounted device that may be mounted in one of thecomputer racks in a data center. In another embodiment, the consolidatormay be implemented using a computer that also performs other functionsin a data center or other facility.

[0033] The intelligent power strip 205 is a power strip having multiplepower outlets and current monitoring devices incorporated within it fordetermining the current draw of any one of the power outlets or thetotal current draw of all devices that are powered from the intelligentpower strip. The intelligent power strip also includes network interfacecircuitry to allow the strip to communicate with the controller 201 overthe network 211. In addition, the intelligent power strip can becommanded by the controller to interrupt power to any of the poweroutlets on the strip. In one embodiment, the intelligent power strip maybe implemented using a Masterswitch VM® power strip, available fromAmerican Power Corporation of West Kingston RI, that has the capabilityof measuring total current drawn by devices powered through the deviceand the capability to control the application of power to individualoutlets.

[0034] In embodiments of the present invention, the controller 201functions as the central controller for the system and communicates withother components of the system over the network 211. In one embodiment,the controller 201 includes a power load monitoring and control module216 that communicates with the other components of the system to controlthe other components and receive power draw levels or current drawvalues from the power monitoring devices. The controller may beimplemented using a single computer contained in one of the racks of adata center, using a desktop computer, a dedicated purpose computingdevice, an embedded computing system, or the functionality of thecontroller may be distributed among several networked computers. Thecontrol module 216 may be implemented using software, hardware or acombination of software and hardware.

[0035] The network 211 provides the connectivity between the componentsof the system. In one embodiment, the network may be implemented usingone of a number of well known network architectures such as an Ethernetnetwork. The network 211 may also be used by the first computer 208 andthe second computer 210 to communicate with other devices within a datacenter or to communicate with devices outside of the data center over,for example, the Internet.

[0036] The system 200 of FIG. 2 may be implemented in the powerdistribution system of a data center as will now be described withreference to FIG. 3, which shows the system 200 of FIG. 2 implemented inthe power distribution system 100 of FIG. 1. As shown in FIG. 3, a powermonitoring device (202A-202M or 204A-204C) is incorporated at a numberof places in the power distribution system 100. Two additional loads 117and 119 are included in the system of FIG. 3. Load 117 is an equipmentrack that contains the controller 201 and the consolidator 206, however,as understood by those skilled in the art, the controller andconsolidator could be located in separate racks or need not be installedin a rack at all. Load 119 is also an equipment rack containing thefirst computer 208, the second computer 210 and the intelligent powerstrip 205. The diagram of FIG. 3 shows only the power connectionsbetween the components. The signal connections between the components ofthe power monitoring system are as shown in FIG. 2. In the embodimentshown, the controller 201 is powered from the power distribution systemthat the controller is monitoring and controlling. In other embodiments,the controller may be powered by a separate power distribution system.

[0037] In the embodiment of the present invention shown in FIG. 3, apower distribution device is placed to measure the current drawn by eachcomputer or server, by each rack, and on each branch circuit at theoutput of the circuit breaker panel. Additional power monitoring devicescould be added at other points in the power distribution system, or ifless monitoring is desired, fewer power monitoring devices could beused. In one embodiment, each of the power monitoring devices may bedefined as having an order value with respect to a given point in thepower distribution system. The order value for a given power monitoringdevice is determined based on the number of power monitoring devicesthat are in the power distribution system between the given device andthe given point. For example, with reference to FIG. 3, for circuitbranch 112 and with the circuit breaker panel 108 as the referencepoint, power, monitoring device 202C has an order of one, and powermonitoring devices 202H, 204A, 204B and 204C have an order of two, andeach of the power monitoring devices in the intelligent power strip 205has an order of three.

[0038] Methods of controlling and monitoring the power and/or current inpower distribution systems using the systems described above will now bedescribed with reference to FIG. 4. However, methods of the presentinvention are not limited for use with the above-described systems, butmay be used with other systems as well.

[0039] In one embodiment of a method 300 of the present invention, whichis summarized in flowchart form in FIG. 4, the method determines themaximum current draw for each of the circuit branches of a powerdistribution system. The maximum current draw can be compared topredetermined values that are based, for example on circuit breakervalues, and if a potential overload condition is detected, warnings canbe generated and corrective actions can be taken. In a first step 302 ofthe method 300, the layout of the power distribution system is enteredinto the controller and a reference point is chosen. In one embodiment,the power monitoring and control module includes a program for enablinga user to enter the layout through a graphical user interface (GUI). Inother embodiments, the power monitoring and control module is capable ofreceiving a data file containing the layout created using one of anumber of computer aided design programs such as, for example, Visio®,Autocad® or a custom designed program.

[0040] Once the layout has been entered and the reference point has beenchosen, in step 304, one of the circuit branches of the powerdistribution system is selected for analysis. Next, in step 306, anorder value is set equal to the highest order value of all powermonitoring devices in the selected circuit branch. For example, if thechosen circuit branch is branch 110 (FIG. 3), then the order value isset to that of devices 202J, 202K, 202L and 202M. In step 308, one ofthe power monitoring devices having the set order value is chosen. Instep 310 all equipment that is powered through the selected powermonitoring device is controlled to draw maximum power either manually,or automatically over the network 212 by the controller 201. Once all ofthe equipment is drawing maximum power, the selected power monitoringdevice communicates a value of power draw or current measured to thecontroller 201 in step 312. Next, in step 314, the equipment coupled tothe selected power monitoring device is returned to its prior state.

[0041] In step 316 of method 300, a determination is made as to whetherthere are any other power monitoring devices in the selected circuitbranch of the selected order that have not yet been selected. If theoutcome of step 316 is YES, then another power monitoring device of thesame order is selected, and steps 308 to 314 are repeated. For circuitbranch 110, there are a total of four monitors having the highest orderfor the branch, and therefore, steps 308 to 314 will be repeated fourtimes until the outcome of step 316 is “NO”.

[0042] If the outcome of step 316 is NO, then in step 318 adetermination is made as to whether there are any devices having anorder value less than the selected device. If the outcome of step 318 isYES, then the set order value is reduced by 1 in step 319, a devicehaving the next highest order value is selected, and steps 308 to 316are repeated. If the outcome of step 318 is NO, then a determination ismade at step 320 as to whether all branches have been measured. If allbranches have been measured, then the process ends at 322. If allbranches have not been measured, then steps 304 to 318 are repeated foranother branch.

[0043] Once all of the maximum current draw values have been determined,the total maximum current draw can be compared to predetermined valuesto determine whether any corrective action should be taken.Additionally, the maximum current draw at each component or element(i.e., a circuit breaker, fuse or other device) having a maximum currentrating can be compared to the current rating to determine if it isnecessary to take any corrective actions. Corrective actions may includeadding a branch circuit, moving equipment from one branch circuit toanother branch circuit, or one of a number of other actions. In a systemin which a UPS is used, the corrective action may include adding anadditional UPS to a branch or adding additional capacity to an existingUPS.

[0044] In the method described above, the power draw on circuit branchesis measured successively. In another embodiment, power draw may bemeasured on multiple circuit branches simultaneously. In thisembodiment, the duration of maximum power for devices may be kept at aminimum to reduce the likelihood of tripping a circuit breaker servingtwo or more circuit branches that are measured simultaneously. As iswell known, a typical circuit breaker will not trip instantly when thecurrent exceeds the breaker's threshold, but typically will only tripwhen the excess current is maintained for some period of time.

[0045] In method 300 described above, the power draw is successivelymeasured at power monitoring devices of lower order. In one embodimentof the present invention, prior to maximizing the power draw of allequipment powered through a particular branch circuit having a powermonitoring device, the total power draw determined using all higherorder power monitoring devices in the same branch is determined. Thetotal is then compared with known allowable maximum levels for all lowerorder devices to ensure that the simultaneous powering of all higherorder equipment at maximum levels will not cause power draw levels inexcess of safe, allowed maximum values. If it is determined that thesimultaneous powering may cause levels to be above allowed maximumvalues, then one of the corrective actions described below may be taken.

[0046] In embodiments of the present invention, one power monitor may bepositioned to measure the power drawn by a plurality of devices withouta higher order power monitor installed between the power monitor and anyof the devices. In such a situation it may be undesirable tosimultaneously bring all the devices to maximum power to measure themaximum power draw. In one embodiment, the devices are operated atmaximum power one at a time with the other devices powered off to obtainthe power draw for each device, and then the individual power draws aretotaled to obtain the maximum at the power monitor. However, in someinstances one or more of the devices may be running a criticalapplication that is intended to be run 24 hours per day, seven days perweek, without interruption. In such a situation, in one embodiment ofthe present invention, the maximum power draw for the combined devicesis determined as follows, using an example of a situation where onepower monitor is coupled to measure the power draw of three devices.

[0047] First, an ambient power measurement is made of the power orcurrent drawn by the combination of the three devices as presentlyconfigured and operating. Next, a first one of the three devices isindividually controlled to operate at maximum power while the other two(the second and third devices) continue to operate at their presentstate, and another power or current measurement is made. Then, the firstdevice is returned to its normal operating state and another ambientpower measurement is made of the combined power or current draw from thecombination. If the ambient power measurement is substantially the samebefore and after the first device was configured to operate at maximumpower, then an assumption is made that the power or current draw of thesecond and third devices remained substantially constant during the timethat the first device was configured to draw full power. The increase inpower or current draw over the ambient value contributed by configuringthe first device to operate at maximum power is then determined bysubtracting either ambient value from the value measured with the firstdevice operating at full power.

[0048] The increase power over the ambient value for maximum power drawfor the second and third devices can be determined in the same manner asthe first device described above. Then the total maximum power can bedetermined by adding the increase for each of the three devices to theambient value. In situations where the ambient value does not staysubstantially constant before and after increasing the power draw of oneof the other devices, then in one embodiment of the present invention,the procedure is repeated a number of times, and if the ambient valuestill does not stay constant, then the ambient value which produces thegreatest increase is used. The increase for each of the three can thenbe added to the worst case ambient value. If the resulting value iswithin acceptable limits, all three devices may then be controlled tooperate at maximum power draw, and an actual measurement with all threedevices at maximum power can be made. If it is determined that thesimultaneous powering may cause levels to above allowed maximum values,then one of the corrective actions described below may be taken.

[0049] The method 300 described above may be performed when equipment isfirst installed, when additional equipment is to be added to a system,or the method may be performed periodically as part of a scheduledmaintenance program. In another embodiment of the present invention, thecontroller provides for constant monitoring of the power draw or currentat each of the power monitoring devices to detect an actual or potentialoverload condition. Present values of power draw can be compared topredetermined limits that are calculated based on previously conductedmeasurements, circuit limitations, or other factors. Rather thanperforming constant monitoring, embodiments of the present inventionalso provide for periodic measurements or scheduled measurements.

[0050] When a potential overload condition is detected, one of a numberof actions, or a combination of actions, may be initiated by thecontroller 201. These actions include sending notifications and loggingproblems as well as taking corrective actions. The notifications caninclude recording an event in a log and activating an audio or visualalarm. Further, the notifications may include sending an email to asystem administrator or facility manager or paging the administrator ormanager. Still further, in some embodiments, the controller may send asignal, such as an SNMP trap, to another computer to notify the othercomputer or its operator of the condition.

[0051] In addition, when a potential overload condition is detected, thecontroller may take positive steps to ensure that an overload will notoccur. In one embodiment, the controller may initiate a shutdown commandof one or more computers by communicating with the computers over thenetwork, or the controller may command one or more computers to operatein a mode that draws less power. In other embodiments, the controllermay also communicate with an intelligent power strip to command thepower strip to interrupt power to one or more of its outlets. In stillother embodiments, all or some of the power monitoring devices include apower interruption mechanism that can be activated by the controllerover the network to interrupt equipment powered through the powermonitoring device. By selectively powering off lower priority devices,the controller can ensure that power continues to be provided tocomputers that are running higher priority applications. Further, newdevices may be prevented from being powered on by automaticallyswitching off the power to outlet strips or individual outlets of outletstrips. After taking one of the above actions to prevent an overload,the controller can determine whether an overload potential has beenavoided, and if not, can take further steps to reduce the power draw.

[0052] In one embodiment, to minimize power draw to avoid an overload,the controller can control a computer device to operate at less thanmaximum load by instructing the computer to exercise a low power taskthat utilizes CPU process time, but during that time, halts operation ofthe CPU. The low power task can be assigned a priority level higher thanother tasks on the computer to ensure that sufficient low power timeoccurs. The average power load of the computer can be maintained at somefixed percentage using this method.

[0053] In one embodiment, the controller is coupled to the UPS 104 overthe network to detect when the UPS has switched to battery mode. Inresponse to detecting that the UPS is operating on battery, thecontroller measures the power draw at the power monitoring devices, andmay take actions as described above to reduce the power draw to minimizethe drain on the UPS to provide power longer for critical applications.

[0054] In some data centers, it is known to provide a dual power feed orsome other multiple power feed to equipment. The multiple power feedstypically provide redundancy and/or accommodate relatively high powerequipment that has multiple power feeds. To prevent overload whenmultiple feed power distribution systems are used, in one embodiment ofthe present invention, the measured maximum current of one feed isadjusted to account for an increase in current that will occur if one ofthe other feeds of the dual feed system fails. For example, in a dualfeed system for a device, in which each of the feeds equally shares thecurrent draw of the device, when determining if a potential overloadcondition exists, the controller multiplies the maximum current measuredon one of the feeds by two to estimate the load on the feed upon failureof the other feed.

[0055] In another embodiment of the present invention, a system 400monitors air temperature at various locations as well as provides thefunctions of the power distribution monitoring and control system 201.The system 400, as shown in FIG. 5, includes all of the components ofsystem 200 plus additional sensors 402A and 402B for measuring airtemperature in a facility. Sensors 402A and 402B are coupled to thecontroller to allow the controller to detect hot spots and takecorrective action. Further two intelligent air conditioning systems 404and 406 are also coupled to the controller 201 over the network 212, andmay also be coupled directly to sensors 402A and 402B. The controller,in response to detecting potential or actual cooling problems cancontrol the air conditioning systems to increase their outputs orredirect their outputs to prevent problems. In one embodiment, the loadsare controlled to operate at maximum power for an extended period oftime, while the air temperature is being monitored, to ensure that theair conditioning system is capable of supplying sufficient cool air forthe maximum requirements. In other embodiments, more or less temperaturecontrollers and air conditioning systems may be incorporated into thesystem 400.

[0056] In one embodiment, the system 400 may further include anadditional sensor, identified as sensor 402C in FIG. 5, located outsideof the facility to detect the outside temperature. As understood bythose skilled in the art, the efficiency of many air conditioningsystems is dependent on outside air temperature. When determiningwhether sufficient cooling is available from the air conditioning units,the controller can account for changes in efficiency of the airconditioning units caused by changes in the outside air temperature.

[0057] In another embodiment of the present invention, which will now bedescribed with reference to FIG. 6, a power monitoring and controlsystem 500 is provided. System 500 is similar to system 400, except thata second network 213 is provided. The second network 213 is used toprovide communications between the power monitors of the system, thetemperature sensors of the system and the controller 201. As shown inFIG. 6, the controller 201, each of the power monitoring devices of thefirst set 202A, 202B, and 202C, the consolidator 206, the intelligentpower strip 205, temperature sensors 402A, 402B and 402C and airconditioning units 404 and 406 are all interconnected by the secondnetwork 213. Also, as indicated by the dotted lines in FIG. 6, inaddition to being coupled to network 211, each of computers 208 and 210may optionally be coupled to the second network 213 in addition to thefirst network 211. In different embodiments of the present invention,computers 208 and 210 can communicate with the controller 201 over thefirst network or the second network or over both the first network andthe second network.

[0058] The second network may be implemented using one of a number ofnetwork types, such as an Ethernet network or a power line carriernetwork. In one embodiment, the second network is a private network thatuses a modified version of the EIA-721 Common Application Standard (CAL)over WP in addition to SNMP and HTTP. The use of the second networkprovides several advantages. First, in embodiments of the presentinvention, the number of devices coupled to the second network isrelatively low, and the amount of data to be transmitted over thenetwork is anticipated to be relatively low. Accordingly, the softwareand hardware required in each of the devices to communicate over thesecond network is not overly complex or expensive to implement. Second,the traffic on the second network is kept separate from the traffic onthe first network, and therefore, the traffic on the second network willnot utilize bandwidth on the first network. In addition, the traffic onthe second network is secure from users of the first network. Thissecurity becomes particularly important for applications in which thefirst network is coupled to the Internet and/or critical applicationsare operating in the computer devices of the network. Another advantageto the use of the second network is that address space on the firstnetwork is not occupied by the devices coupled to the second network.

[0059] In above-described embodiments, external power monitoring devicesare used to monitor the power to computers or groups of computers. Inanother embodiment, some or all of the computers may have powermonitoring devices contained within, allowing the computers to monitortheir own power, and directly report their power draw to the controllerover the network 211.

[0060] Embodiments of the present invention described above are for usewith AC power distribution systems. However, the present invention isnot limited for use with AC power distribution systems but also may beused with DC power distribution systems. In addition, embodiments of thepresent invention may be used in data centers that utilize both AC andDC power distribution systems. As understood by those skilled in theart, when used with a DC system, several components of the ACembodiments described above may not be needed, such as a powerdistribution unit containing a transformer.

[0061] Embodiments of the present invention described above, overcomeproblems associated with designing and maintaining power distributionsystems in data centers by providing more accurate monitoring andcontrolling capabilities of the power draw of computer systems coupledto a power distribution system. In embodiments of the present inventiondescribed above, computer systems are controlled to operate at 100% oftheir maximum power to calculate the maximum power draw on circuitbranches. As understood by those skilled in the art, in otherembodiments, computers could be controlled to operate at knownpercentages of full load (i.e., 50% of full load and 75% of full load)and scaling factors could be used to extrapolate full load values basedon measurements at known operating points. Such a system is advantageousin that it may be safer to first operate a device at a known value lessthan full power to determine if any problems may occur at full powerbefore operating the device at full power.

[0062] In embodiments of the present invention described herein, currentmonitors are used to measure the current drawn by a given device or agroup of devices to determine whether maximum current or power valuesmay be exceeded in a system. As understood by those skilled in the art,the power drawn by a device is related to the current drawn by thatdevice, and embodiments of the present invention are not limited tosystems that utilize current monitors, but rather, also include systemsthat utilize monitors based on power and/or other electricalcharacteristics.

[0063] In embodiments of the present invention discussed above, anadditional task having low priority is added to a task list of acomputer to cause the computer to operate at maximum power. In otherembodiments of the present invention, a task having a high priority maybe added to cause a computer to operate at a predetermined percentage ofmaximum power. The task that is added may cause the processor to be idleor to operate at maximum capacity depending on whether it is desired tooperate at a low or high percentage of maximum power. For example, inone embodiment, a computer can be controlled to operate at a minimum (orambient) level by causing the processor to be idle for nearly 100% ofthe processor time by using a task that has a high priority, requiresmaximum processor time, and places the processor in an idle state. Bycausing a computer to operate at the ambient level and then the maximumlevel, the power consumption dynamic range of a computer can bedetermined. This dynamic range may be used by power distribution systemdesigners in designing facilities.

[0064] Embodiments of the present invention are described above as beingimplemented with rack mounted computers. As known by those skilled inthe art, in some data centers, computer servers are implemented assingle cards, identified as server blades, installed within a commoncard cage or chassis, which is in turn typically installed in a rack.Embodiments of the present invention may also be used with server bladesto individually control the power draw of each server blade and tocontrol the combined power draw of two or more server blades installedin a common chassis.

[0065] In some embodiments of the present invention, as described above,the control modules 212 in the computers 208 and 210 are described asbeing implemented by software or a combination of hardware and software.In one embodiment, the control module is implemented as software that ispackaged and installed with UPS management software.

[0066] Having thus described at least one illustrative embodiment of theinvention, various alterations, modifications and improvements willreadily occur to those skilled in the art. Such alterations,modifications and improvements are intended to be within the scope andspirit of the invention. Accordingly, the foregoing description is byway of example only and is not intended as limiting. The invention'slimit is defined only in the following claims and the equivalentsthereto.

What is claimed is:
 1. A system for monitoring power in a powerdistribution system, the system comprising: a power monitoring devicelocated in the power distribution system to measure a value of at leastone characteristic of power provided to a branch of the powerdistribution system, the power monitoring device having an output thatprovides the value measured; and a controller having an input to receivethe value measured and an output that couples to a first device poweredby the branch of the power distribution system to send a power signal tothe first device to command the first device to operate at apredetermined percentage of maximum power.
 2. The system of claim 1,further comprising a plurality of power monitoring devices, each locatedin the power distribution system to measure at least one characteristicof power provided to a respective branch of the power distributionsystem, and each having an output to couple to the controller to providea value of the characteristic measured.
 3. The system of claim 2,wherein each of the respective branches of the power distribution systemprovides power to at least one respective device, and wherein thecontroller is adapted to send a power signal to each respective deviceto command each device to operate at the predetermined percentage ofmaximum power.
 4. The system of claim 3, wherein the controller isadapted to send the power signal to devices powered by one branch at asame time, to cause each of the devices on the one branch to operate atthe predetermined percentage of maximum power.
 5. The system of claim 4,wherein the controller is adapted to total the values measured for eachof a plurality of branch circuits and compare the total with a firstoverload value to detect an overload condition.
 6. The system of claim5, wherein the controller is adapted to send an alarm signal to anoperator upon detection of an overload condition.
 7. The system of claim6, wherein the controller is adapted to send a signal to disconnectpower to one or more devices upon detection of an overload condition. 8.The system of claim 7, wherein the at least one characteristic iselectrical current.
 9. The system of claim 8, wherein the controller hasa first network interface to communicate with devices powered by thepower distribution system over a first communications network and asecond network interface to communicate over a second communicationsnetwork; and wherein each of the plurality of power monitoring devicesincludes a network interface to communicate with the controller over thesecond communications network.
 10. The system of claim 9, wherein thepower distribution system includes an uninterruptible power supply, andwherein the controller is adapted to communicate with theuninterruptible power supply to detect that the uninterruptible powersupply is operating on battery mode and replace the first overload valuewith a second overload value.
 11. The system of claim 10, wherein thecontroller is adapted to send a signal to interrupt power to at leastone device upon detection that the uninterruptible power supply isoperating on battery mode.
 12. The system of claim 11, furthercomprising a plurality of temperature sensors that monitor temperatureat locations within a facility, each of the temperature sensors havingan output to communicate a temperature value to the controller.
 13. Thesystem of claim 12, wherein the controller is adapted to comparetemperature values received from the temperature sensors withpredetermined values to detect an over temperature error condition, andupon detection of an over temperature error condition sending an alarmsignal.
 14. The system of claim 13, wherein the controller is adapted tosend a signal to interrupt power to at least one device upon detectionof an over temperature error condition.
 15. The system of claim 1,wherein the controller is adapted to compare the value measured with apredetermined value to detect an overload condition, and the controlleris adapted to send a power interrupt signal to interrupt power to thedevice upon detection of an overload condition.
 16. The system of claim15, wherein the controller is adapted to control at least one device tooperate at a reduced power level upon detection of an overloadcondition.
 17. The system of claim 15, wherein the power monitoringdevice has a plurality of outlets, and the power monitoring device isadapted to send a signal to the controller to indicate a value of the atleast one characteristic for each of the plurality of outlets.
 18. Thesystem of claim 17, wherein the power monitoring device is adapted tocontrol each of the plurality of outlets, and in response to the powerinterrupt signal from the controller, the power monitoring device isadapted to interrupt power to at least one of the plurality of outlets.19. The system of claim 1, wherein the at least one characteristic iselectrical current.
 20. The system of claim 2, wherein the controllerhas a first network interface to communicate with devices powered by thepower distribution system over a first communications network and asecond network interface to communicate over a second communicationsnetwork; and wherein each of the plurality of power monitoring devicesincludes a network interface to communicate with the controller over thesecond communications network.
 21. The system of claim 1, whereinpredetermined percentage of maximum power is 100 percent.
 22. The systemof claim 8, wherein the predetermined percentage of maximum power is 100percent.
 23. A method for monitoring and controlling a powerdistribution system that has a plurality of circuit branches forproviding power to a plurality of devices, the method comprising:controlling a first device on a first circuit branch to operate at apredetermined percentage of maximum power; detecting a first value for acharacteristic of power provided to the first circuit branch;controlling a second device on a second circuit branch to operate at apredetermined percentage of maximum power; detecting a second value fora characteristic of power provided to the second circuit branch; addingthe first value to the second value to obtain a total value; comparingthe total value to an overload value to detect an overload condition;indicating an alarm condition when the total value exceeds the overloadvalue.
 24. The method of claim 23, wherein the first device iscontrolled to operate at less than the predetermined percentage ofmaximum power when the second device is controlled to operate at thepredetermined percentage of maximum power.
 25. The method of claim 24,further comprising controlling one of the plurality of devices tooperate in a reduced power mode upon detection of an overload condition.26. The method of claim 25, further comprising interrupting power to oneof the plurality of devices upon detection of an overload condition. 27.The method of claim 26, wherein the characteristic measured iselectrical current.
 28. The method of claim 27, further comprising:communicating with the first device and the second device over a firstcommunications network; and communicating with power detection devicesover a second communications network.
 29. The method of claim 27,wherein the power distribution system further includes anuninterruptible power supply, and wherein the method further includessteps of: detecting when the uninterruptible power supply is operatingin a battery mode; and controlling at least one of the plurality ofdevices to operate in a reduced power mode upon detection of the batterymode.
 30. The method of claim 29, further comprising interrupting powerto at least one of the plurality of devices upon detection of thebattery mode.
 31. The method of claim 30, wherein the power distributionsystem is at least partially contained within a facility, and whereinthe method further includes steps of: measuring air temperature at aplurality of locations within the facility; comparing at least one valueof air temperature measured with a predetermined value to detect an overtemperature condition; and controlling at least one of the plurality ofdevices to operate in a reduced power mode upon detection of the overtemperature condition.
 32. The method of claim 31, further comprisinginterrupting power to at least one of the plurality of devices upondetection of the over temperature condition.
 33. The method of claim 26,wherein the power distribution system is at least partially containedwithin a facility, and wherein the method further includes steps of:measuring air temperature at a plurality of locations within thefacility; comparing at least one value of air temperature measured witha predetermined value to detect an over temperature condition; andcontrolling at least one of the plurality of devices to operate in areduced power mode upon detection of the over temperature condition. 34.The method of claim 33, further comprising interrupting power to atleast one of the plurality of devices upon detection of the overtemperature condition.
 35. The method of claim 23, wherein the firstdevice and the second device are controlled to simultaneously operate atthe predetermined percentage of maximum level, and the method furthercomprises steps of: measuring the combined maximum power draw for thefirst circuit branch and the second circuit branch; and comparing themeasured combined maximum draw with the total overload value.
 36. Themethod of claim 23, wherein the characteristic measured is electricalcurrent.
 37. The method of claim 23, further comprising: communicatingwith the first device and the second device over a first communicationsnetwork; and communicating with power detection devices over a secondcommunications network.
 38. The method of claim 23, wherein thepredetermined percentage of maximum power is 100 percent.
 39. The methodof claim 27, wherein the predetermined percentage of maximum power is100 percent.
 40. A system for monitoring and controlling a powerdistribution system that has a plurality of circuit branches forproviding power to a plurality of devices, the system including: meansfor controlling each of the plurality of devices to operate at apredetermined percentage of maximum power; means for detecting a valueof a characteristic of power provided to each of the plurality ofcircuit branches.
 41. The system of claim 40, further comprising meansfor comparing a total value of the characteristic with a predefinedvalue to detect an overload condition.
 42. The system of claim 41,further comprising means for interrupting power to at least one of theplurality of devices when an overload condition is detected.
 43. Thesystem of claim 42, wherein the characteristic is electrical current.44. The system of claim 43, wherein the power distribution systemincludes at least one uninterruptible power supply, and the systemfurther comprises means for detecting that the uninterruptible powersupply is in a battery mode of operation, and means for adjusting thepredefined value when the uninterruptible power supply is in the batterymode of operation.
 45. The system of claim 44, further comprising meansfor detecting air temperature values in a facility containing the powerdistribution system.
 46. The system of claim 45, further comprisingmeans for comparing the detected air temperature values withpredetermined temperature values, and means for interrupting power to atleast one of the plurality of devices when the detected air temperaturevalues exceed the predetermined temperature values.
 47. The system ofclaim 40, wherein the power distribution system includes at least oneuninterruptible power supply, and the system further comprises means fordetecting that the uninterruptible power supply is in a battery mode ofoperation, and means for adjusting the predefined value when theuninterruptible power supply is in the battery mode of operation. 48.The system of claim 36, further comprising means for detecting airtemperature values in a facility containing the power distributionsystem.
 49. The system of claim 48, further comprising means forcomparing the detected air temperature values with predeterminedtemperature values, and means for interrupting power to at least one ofthe plurality of devices when the detected air temperature values exceedthe predetermined temperature values.
 50. The system of claim 40,wherein the characteristic is electrical current.
 51. The system ofclaim 40, wherein the predetermined percentage is 100 percent.
 52. Thesystem of claim 43, wherein the predetermined percentage is 100 percent.